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achievement, ^researchers reanalyzed the dita used in "Public 'and 
Private, Schools" (Coleman et al.) at the school^ level rather than the 
individual ^ level . The data for the Coleman report and the present 
reanalysis are drawn from the "High School' and Beyond" study, a 1980 
national ^u^vey of 30 sophomores in each of 1,002 high schools. The • 
researchers find that using regressiTon equations on aggregate 
school-leVel data instead of incji vidual-level data reduces the error 
in individual-level equations' (caused by error in variable 
measurement ) and controls for the contextual effects of student-body 
§ocioeconom3?!?'^^tus. The reanalysis indicates that the apparent 
superiority of'private schools in academic achievement • is much 
smaller when computed at. the school level rather than, at the 
individual level. (Author/RW) * 



********************************** 

* Reproductions Supplied by EDRS. are the best that can be mkde * 

* from the original document. \ * 
********************************** ****************************^**L ****** 



ERLC 




U S DEPARTMENT Of EDUCATION 

NATIONAL INSTtTUT^OF EDUCATION 



Report No. 323 
March 1982 

ACHIEVEMENT PREDICTION WITH SCHOOL LEVEL EQUATIONS: 
A NON-TECHNICAL EXAMPLE USING THEJ PUBLIC AND 
PRIVATE SCHOOLS DATA 
Robert L. Crain and Robert L. Ferrer 




Intjoductory Statement 



IJ^pvm'"" Organiz-ation of Schools has tw^ primary objectives- 

nd r :his"L"'f'H °^ how -schools affecftheJ^ s ^ ^ s 

tion' knowledge to develop, better school practices and organiza- 

S^udJesI; Sch'So'l'n'°"'' fiv-progpams "to. achieve its ob>.tWs. The 

such as housing and ioB deseerp^>o^^ nr» tuI c u i <^ . xbbue-s 
is currently conce.n^-d wi^ra^^r^^-IconTrol^cI^^ 
rod cedT! rL'Tl ' ^"'h^^^? S-up -processes in' schools It'Ls 
Studenf Teai Larn ni'- ; ' °' '''^^'^ °' ^^^ools, has developed 

dec'r n eWnta?v and "^'^ P«-ocesses for teaching various^sub- 
sysfem ?or schooflidf ' "-^-Produced a coi;,puterized 

Ca'eer Delel^lZnl ^"endance monitoringr,. The Schpol Proces^and 
SifSSS^t>°r" " f "'"'^^ transitions f r-S^ThlihTHh^-.o ' 
d^caSr plaL aL he"^ °' schooling in the. development 

Studies in DM? actualization of labor market outcgmes, T^e 

charac eris JcJ In ^^J^^^,'^"-^'^— ^s , school -exp-e.iences , and ^'ividual 
acteristicis in relatidn to in-school ajid later-life del/nquehcv 



th1t';ro"Ide1^;;:??:n":ie^^f~^^ X 
and publish qiPn?fTn.!i ' young researchers to conduct \ 

than ai the .individJai Jevel ' " ^^'^"'^ ^'^'^ 



- r: 



Introductory Statapient 



to deX^f° °^Sch■ools has tw. primary ob/eccives: 

and to use this kno^f h of how schools affect their students, 

tion' knowledge to develop better school ^Vactlces and ojrganLa- 

S^udJef r^^^-^J'^^^gh fWprograms "to. achieve its ob>sWs. The 
■ ^^i^i^h^^^f''^^ ^PPl^- ^he ba.ic^ es of %, 

•deseJregafeS school "^^'r''-'" ^'"""^ ^"^^^^^ conditions of ■ 

an fhe'i^Lr uSo'^s of sc'h"'' 'h'^' °' alternative desegregation policies 
• such a. ^ouX lT^^^^^^^^^^^^ ^^-'^y i--ue. 

Ju4--::;L-;s-r "^^^^ 

Lures, reward systems, and peer group -processes in" schools It hp^ 

udtfTe\i\^^;:jn?-"r'^ °' ^^^'^"^"^ °^ open^^to!;, jL'^^veloped 
•jLtrin SLeSaJv !■ H Processes for teaching various sub- 

ys L "ofXoSide atr°H "--produced a cof^puterized 

Career Deve^nn^^i/ attendance monitoring.. The Schpol Proces^and 
^^i^^fl^.llT"'' r^'^'^S transitions fr-6^hiir7^h^-,o ' 
of^areer pLL and th^""^ °' schooling in the. development 

Studies in nf?J actualization of labor market outcpmes. 'de 

characteris iL In -"^s , school -experiences, and ^'ividual 

acteristics in relation to in-school and_ later-iife deli^nquehcy . 

lTJTrT',/^^° supports a Fellowships i n Education Research' prograrf 

and pub- Lh s'i^niH '""j.'^'^^' '"'^ t^entedT o^g-Tiii^rs-t'eond^^ \ 

L^'LIJzL^'^the'^pSbUc 'Z^'l^^T^^'^'i ^--^-^ation prog;am, .'■ 
than at the .LdiwLi Uvel ' ' '^'""'^ ^'"'^ ^-^^ -^"er 



ABSTRACT 

m 



' Analysis of the effects df school, .traits ori achievement done with 
aggregate school-level regression equations reduces considerably the 
error appearing in individual-level equations due to error in the measure- . 
ment of vaViables," especially in the measurement of studen social status. 
Aggregate-; evel equyiontf also control on the contextual e,fect^of student 
body SES. Aggregate equations are often the logically correct level of 
analysis, with a more plausiWe number of degrees of freed, m. A reanalysis 

V I ' * 

of theS^a from the Public and Private Schools report finds that the 
apparent superiority of private schools (altho'ugh- still overestimated) ' 
is much smaller when coinputed with school-level equations < h?n when 



estimated at the individual level, 



I 
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. • It is common practice in educational sociology to compute regression- 
equations in order to isolate the effec't of school, characteristics , net- ' 
of the background of students, "ihe most commort use is to isolate the 
effects of some scKool factor in the achievement test secures of students\ 
Unfortunately, it is also common practice ta criticize such studies on 
two grounds; (i; tl.e measurement of 'control variables, whether they are 
pretest scores of achieveme'nt o'r measures of family sociaeconomic status, 
are. made with error. Att^entuation of the regression coefficient* will 
•have the effect of overstating the iipacf of a^ny school characterise ' 
vhich is positively correlated with pretest scores or SES.^ (2rlnaly^is 
must control not only on^student SES, but on the .cantex£yal effect 'of ^the 
SES of other students.^ As Coleman el aX. (1966) dem^onstrated , the school 
student body social class can be quite strongly related to individual 



X 

student achievement. 



' -One partial solutfon to these problem^' is to sim'ply compute' regression 
equations with "data- aggregated to the school level. Ther.e are several 
advantages to a schdal -level analysis: 

(!•)' For many such analyses the school is the, logical unit'o'f analysis- 
Often,' resWch is intended to determi'ne whether schoqls containing some 
particular charac telristic are superior learning environments compared to 
^^oqls which dp not have^this characteristic. For te'sts of statistical^ 
slgrjiFicance, the .cbrrect nUer-of degrees of fteedom in* such an argument, 
is the number of schools , Involved , ^lot 'the number of students. This is no7 
always' the ^cas.^.^^ tor ex^mpU, if "one is evaluating 'an- experimental method' 
of ^teaching, applied Un one'g'chool with a second school/used as control ' 



n 
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group, it is- appropriate to-set the degrees of freedom equal^Che ^ 
number offstudents involved^ But in this case the- object of the experiment 
!is'to demonstrate whethe^a particular treatment as executed in a particular 
school is superior to a normal 'situation. If is weU understood that such 
Research cannot demonsfite that the application' of. the same, treatment in • 
another school, with another set of implemeiTt|tion characteristics,, 'would 
produce the same result. Once an .education'St' innovation has gottk^ past, 
the experimental stage, most research asks a different .question : What is 
the overall effect of this innovation, as implemented in a t^ariet}4of 
settings? This question can only be answered by drawing a sample of learning 
, environments (either classrooms or .schools) which have this characteristic 
apd contrasting kem to a control sample of learning environments which 'do 
not. do not mean to oversimplify the sometimes complex question of 

assessing the number of degrees of freedom in ^ research design, but only 
to point out, that in many cases^ the number of degree? of . freedom is the 
number of learning environments, not the number of students. . . 

(2) Aggregation of data to the school, l^yel is necessary in order 
to compute student body socioeconomic status oiq average pr-test ax:hieve- 
ment for use as a'control variable. Admittedly, this aggr "gated character- 
'istic of "the Classroom could be attached to ind'ividbal student rfecords 

for an individual level analysis, but e^^^n s,p , aggregation ^is a necessary 

.prior step. ' ' » " 

(3) .The aggregation of data to the learning environment level reduces 
the error of measurement in pretest achievement scores or socioecanomic - . 
status. Given economic segregation and segregation of students by ability, 
the 'ability or family SES of the other students in a classroonv is corr|la-teti 
with .an individual's ability or SES. Thus adding the composite .classroom ' 



score on a" variable increases th^ apcuracy with which *an individual's 
•score is measured. For thi(s reaso'n , it is epctrao'rditiarily difficult to ' 
^ separate the individual effects of SES on achievement from the classroom 
•contextual ef fee t-s . .because what appears to be classroom contextual effect 
. are partly merely the correction of <neasurement error in the individual 
students' SES. • ■ " " , 

In this paper, we will demonstrate the differences in the results ' 
obtained -from individuk-le4l ^nd school-level analysis usipg data fro^ 
■ the National Opinion Research Center's Hi^A. SchooL^n_d_Bei.ond- survey, 

with an analysis sii^lar to that conducted by Coleman, Hoffer, anM Kilgore 
-"(1981). Their analysis ^ttemj^ted to sho^hat net of family background 
factors. Catholic and* non-Catholic private schools produce highe^r student 
achievement than do public schools. They used several analyses,' but all* 
had been criticized for three e;rors:2 inability to control for 

self-selection bias, which might" result from either the higher motivation 
of students attending j,rivate schools, or the selection criteria used by 
private schools to determine which students .^/ill attend; (2) 'failure to 
correct for attenuation of the regression qf achievement on social class. _ 
'variables, which would tend to overstate the achievement benefits / 
attributable to the private schools, whibh hdve higher SES students-,' 
(35 failure to remove the contextual effect^ of sopial class at the same 
^time that the individual level affects were controlled 

Aggregation of the data to the school level can do little to correct 
fqr self-selection Mas. An ana^s of the actual admission criteria ' 
of private ^scho^ls and^ ideally^ e)^perlment in which students were 
randomly assigned to public or private schools seem to be the only 
techniques which could deal with this is6ue . Because of tnis problem, 
some critic^ have concluded that it is impossible to dsse\s the relative 



quality (measured in achievement outcomes) ofpublic and private schools, 
An aggregate analysis can, however, incorporate the contextual effects of 
;?,cial class and can reduce th, amounti of error in 'the measuremen^ /f ^ . 

f 

family background. * . * ' * , . 

self-selection -bias, measurement ertor, .and failur^e to include con- 
textual SES effects all work to overestimate private school quality. No ^ , 
major biases in the individual-level data work in the- opposite direction, 
so the individual-level regression result's .are estimates ofyhe upper limit " 
of the. effect of private schools. An aggregate analysis does little to 
correct^o;. 'self-selection bias, and does not eliminate all measurement 
error, but it should -provide a lower value for this estimate of the upper^ . 
limit of the private/school effect.- 

The school is also the logically correct unrit of analysis. Thi fact 
that 30,000 sophomore students were surveyed is misleading. Cronbach • ^ 
(1981) po-inted out that <here are only 27 «on-Catholic private schools in 
'the sample and h^ argued that this is too small ? sample to draw any _ 
' conclusions about a very heterogeneous pool of schools. The fact that 
over 500 students were suweyed in th^ese schools is irrelevant. ^ 

• fhe^Data 

^ Hl^h_School and Beyond is a survey dj>ne in the Spring of 1980, of 1 
30 so'phomore students in each of 1,002 high schools, a stratified sample . 
representative of ^the^ United States .^an^ly-s7 students are weighted 
^to create a sample ;epresentative of national population of spphomores. 
Added to the -stratifiednsam^le are data from the 10 academically most 
successful private schools in the United States,^ At the same^Ume, seniors 
in e«ch school were also surveyed, and plans are to follow both cohorts _ 
for an indefinite period of- time to analyze post-high school adjustment 
to ca^lege and work. The study is in many ,ways a replication of the earlier 



National Longitudinal Study of the High School Class of 1972. Data are 
from self-administered. achievement tests and questionnaires, and are Z 
supplemented by a'survey of the >chooi 'piiinqipals . Data frcfli, the. principal 
.are not used in the analysis that follows. 

^ T he) Original A nalysis ' ' ■ 

, Achievement test scores ark considerably higher in Catholic, and non- 
Catholic private schools than they are in public schools. Howevej:, SES 
differences are also large, so it is nonappropriate to simply concluSe - 
that private schools are providing' a su^ior educatidnal environment. 
One of the analyses conducted by CoJ^an, Hoffer, and Kilgore, and the one 
which, has rec^ved the gredtest^attention, is a cross-sectiona-l regression 
0/ the sophomore sample.^ Regress:^on equations^ were computed separately 
for public school students and private school students ,^^using a large number 
of, family background variables as predictors of achievement. Analysis was- 
done for three achievement tests:' vocabulary, reading,^and mathematics. 
In their report, Coleman, Hoffer, and Kilgore ^'ompute '.the expected achieve- 
meat outcomes for public school students which one would obtain by s&b- 
stituting the means for public school student/s into' the regression equations 
computed on private schools. The^ d'if.f erences\etween these expected 
values and tlie actual* values .of students in public' school are a measure 
•of the effectiveness J private schools. An additional dummy variable 
was added to separate Catholic f rom'non-Catholic schobls, sd that 
separate estimates for each could bp' made . ' • " ' - 

' - ' 

. 4 The Reana lysis 

In our.reanalysis, we simplified somewhat the procedure used in the 
public and private schools' report. Rather^than running separate equa- 
tions for^public schools and private schools, we ran asir^e regression 
equation for the entire population, usirig dummy variables fo is^l'ate the 



effects of catholic 'and noit-Catholic porivate schools. We used afe our 
measures of family background the same variables used in t>e original 
report. Because that - list, wa's itself ' 'derived, from a larger list one 
might expect that a slightly better equation cbuld be constated to fit 
the achievement data for the* pooled- public-private school sample, but we ^ 
assumed- the differences would be sm^ll and were' interested in staying as 
close to the original technique as ^possible. There is no reason to expect 
^arge differences in the estimates of private schoo; effects obtained 

from'the pooled 'regression equation and those obtained by substituting 
.mearis from public school equations into 'private schbol equations, .and^as ^ 
we shall see later, (in Table 2) the, differences are small. 

We thl c'omputed the same regression equations with aggregate^data .- 
Here the dependent variable, rather than being the achievement of a singl^ 
student, becomes the mean achievement of all the student^ sampled -in'a^ - ^ 
particular school. Similarly, the individual father's education' is ^ 
replaced by the mean fathers ' .education 4«f all students in the school, and 

on- If there were no error of measurement and no' school contextual 
effiectsy the unstandardized regression coefficient would be identical .in 
the.aggregate aftd individual analyses. The standardized coefficients 
;ould differ, however., because the stldard Aviations of t^e independent 
and dependent variables would be different at, the schooi. level and^ 
individual level. Generally . the school-level b.etas, will be higher . But 
if a variable 'has measurement error. <<r if there is a contextual effect 
in^the same direction as the individual effect.^ then the unstandardized- 
regression coefficient will also be higher at the aggregate leve^l. To - 
dlmons^rate- this in a simplified fashion. .Table 1 shows' individu'al and 
aggregate equations using oply threa variables. The achiev.ement outcomes 
are the number of correct answers (corrected for guessing^ on tests of ^ ' 

r ' 
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vocabulary, reading, and matHematics . " The first, six 'lines ' of the table 
apply to the'vocabulary subtest. Thg first two lines show individual and 
aggregate equations using only the CathoAic and other 'private school 
dummy variables as independent variables, because"*- th«e ownership of the 
school is a ^chooh-leve-l variable, the ' individual and aggregate analyses ' 
must be^ide'ijtlcal. _ They differ" slightly > this case .because'of differences 
in the way missing values^^re handled at the individual and aggregate level, 
Although the unstand^rdized coefficients are almost the same, the multiple ' 
R is considerably higher at the aggregate, as expected. In the third and 
^ fourth lines we add a single measure "of fa^nily background-father's educa- 
tion-and the t^equations diverge sharply. School mean father's educa- 
tion as a- predictor of aggregate student achievement has an unstandardized 
'^•regression coefficient over twide as large as the parallel individual-level 
coefficient. Because Cathodio and other private schools l^ave higtter mean 
father's educations, a stronger effect of fiber's education tends to 
reduce the apparent superiority of privaTeV schools . In *his case the ' 
, coefficient for Cath61ic schools, falls to slightly over half of the ., 
individual level coeffic^ient and the coefficient for' other private .schools 
I fills- to less than one-fifth of . the individual level coefficient. In the 
fifth ^and'sixth lines of the table we demonstrate the' overall- effect of 
controlling, on father's education by looking at the ratio^of the regression . 
^coefficients for Catholic and other private schools to the regression 
coefficien^ts for^these two variables before father's education enters the 
equation. At the individual levSl. the coefficient for 'Catholic schools ' . " 
is slightly less than 3/4 of , its' uncontrolled coefficient, and for other 
private schools dhe ^efficient for" Catholic schools drops to 3/8 o%the 
.uncontrolled coefficient and the coefficient fbr other private schools ' 
is only 1/lOth.of the indiVidua;-level coefficient.' In the ' remaining • 
two panels of the table we see'a very^ ^imilar^atte^n . . Before father's 
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education Is entered the individual and aggregate coefficients for^hool 
o«.crshlP are very si.'^,. After fa'ther's, education Is entered, the school 
o^ershlp coefficients are .uch lower at the aggrigalle level .than at the" 
individual levil! dropping Wost t, zero for the other private 'a^ihools . 
in all. cases the multiple ^relation coefficient Is Mgher at the aggregate 
. level than at .the individual level. 




Table 1 About Here 



. Table 1 demonstrates that the aggregate-level analysis works as . 
expected. In Table 2 , we _ replicate the Coleman, Hoffer, and Kilgore 
analysis, using the eighteen background variables that they selected. 
They selected these 18 from a larger po^l of variables, choosing thos# _ 
which entered the equation%n the expected direction. If we wanted to 
obtain the best " possible pooled individual level equation or the best 
possibl'e aggregate-level pooled equation, we should make our own selection 
from the- larger pool. .However, we want to compare this analysis to the 
• original .nalys'is, we we'have retained the original 18 variables. Looking 
' first at the individual level analyses,' we see -patterns . similar to tl^g^se 
EQund in the Caleman, Hoffer, and Kilgpre analysis. The apparent ■ 
•superiority of Catholic schools is maintained,, and the coe^ f icient-s are 
close tojhe .stimate^'obtained in the original analysis. This is shown 
.i. the second 4nd fourth lines from the bottom of t^e table, witich give . , 
the ratios of the controlled, and uncontrolled regression .coefficients for 
, catholic" schools, first from these equations and then from the equa^tions" 
- used.^by Coleman,- Hoffer, and Kilg/re; 'Tor all three subtests, the ratios ■ 
are similar, although always slightly larger^n"th. origin,.l analysis. • 
Apparently tfle' reason fo; this i. that black students in C .thoUc schools 
tend to have unusually high achievement. Hence, when the- public school 
means .re substituted into pkvate school equations, the effect ^ a Urget 
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* number of black students in the public schools te 
overall achievement very much. In the pooled analysis, the achievement 
of black students is largely determined by the per^^^e of blacks in 

. public ^fchools, 'Since the vast majority are in public ^ools, so the 
regression slope for rape is'ste^^er than in Coleman , ^Hoffer and Kilgore's 
analysis, and somewhat more of the public-privafie school difference is 
removed. The first and third line4 f^gm-the bottom show 'Ihis pattern 

• for^'other'priva^te schools. Here ke differences are somewhat greater, 
especially in reading, where the original ^alysis fkds a sizea^e ^ 
positive effect on attending other 'private schools .and the present 
individual- level analysis shows very little. | ^ 



Table 2 About Here 



The original analysis reports no significance tests. Obviously, the 
^,000 surveyed students do.not represent 30,000 independent trials. 
Formulas to estimate statistical significance from a weighted cluster 
sample could be used, but ve -have,. not' done ^ . To give, a rough estimate ' 
of significance, we have assumed that each school contributerequally to 
the analysis (i.e., we ha^e ignored the school weighting), and have assumed 
^that eaA of tl,e 973 schools with achieyement^data present is an ^pdepen- 
dent triar. With^973 degrees of freedom in the analysis, there are eight 
family background variables significant in each equation; six of these 
are the same for each .subtest. Of thV fifty-four regression coe-filc_ients 
for the family background variables in the three individual level analyses, 
only four coefficients have entered in the unexpected direction (for / 
example, father absence seems to increase vocabulary score r)theK than . 
decrease it). However, none of these four coefficients are significant. 
Assuming 973 degrees of ^edom, the effects of attending non-Catholic 



private schools and Catholic schools are not significant in any of the 
equations. ^ ' 

At the aggregate level, the family background factors enter the 
^equation in somewhat the same .manner , but moSt of the coefficients are 
larger, Reflecting t^Q reduction in error and the. increase due to the 

addition of contextual effects. From nine to twelve variables aresigni- 
^ < 

ficant in each equation. The effects of Catholic ^nd non-Catholic 

# 

schools are reduced sharply.^ For non-Catholic schools, all three co- 
efficients are negative, but not signif icanJt . For Catholic schools, the 
coefficients drop to less than h^j^f . of their values in the individual- 
level equation, but twg of tbese three coefficients are now statistically 
significant because of the smaller between-school variance which is being 
analyzed. • ' - 4 

In the ori*ginal analysis, the use of a large niimbftr of family back- ^ 
ground Variables was intended to provide the^best possible estimate of 
an ove*^ll family backgrooind effect in order to arrive at the least biased 
estimate of the effecjis of private schools. Because there is a great deal 
of 'multicollinearity in the control variables, the . coefficients for 
individual variables var^^ considerably across the ' six equations. For 
example, the coefficient relating mother's aspirations for. the student 
to math performance is twice as high as th'e coefficient for the same 
variable for the vocabulary test, but we doubt that' this difference is 
interpretable. Similarly, the' percentage of students with typewriters 
at home is strongly correlated with aggregate vocabulary t^st score, 
\>7hile individual typewriter ownership is not strongly related tb individual 
vocabulary score, but again we doubt that a substantive interpretation is 
possible from this analysis. There may be substantive differences between 
the equations which might be interpreted in an analysis intended for. that 
purp'ose, but that interpretation of these particulai coefficients in this 
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set of equations seems unwise.) ' - 

Earlier we argued that the biases of self-selection, measurement error, 
. and the absence of contextual effects all serve \o overestimate the effects" 
of private schooling'. ' The aggregate analysis has r^educed but not eliminated" 
measurement error -and^itm ignores self-selection bias, so we belie^f . 
the privatre school effects are still overestimated, and thifik the aggregate 
coefficients should be viewed as upper limits, ^rather thap unbiased, 
estimates. Thus the main conclusion of the aggregate analysis is that the" 
effects of private schooling probably do not- exceed the values obtained 
in the aggregate equation, which are in turn considerably smaller than those 
_ . obtained in WCindiv^ual l.evel . Coleman,' Hoffer and Kilgore argue that • 
by including a large -^^umber of individual family background variables, they 
■ . may have oyerestimatfed the effects of SE^ . Fo^^ example', parental aspirations 
may. have been heighte'fied by the performance of their child in private sthool, 
so that parental as?4.rations cannot be viewed as a prior variable , in the 
analysis. It would b^e^.p'gssible to test this 'argument by runnirTg a 
variety of equatiojis Ileleting various varial^^es. _ 

. Figure 1 plots th'e^mean reading acl^ievement of each school against 
. the best. prediAor or achievement for that school-the linear combination 
. of independent variables ' generated by the Wgression equation . With a52 
: public schools, it would be difficulty to read the pl^t, so we have 
•simplified the drawing'^by omitting Mt of the public school dita points. 
- Instead, we have drawn a topographic map of the 'data. The small closed 
curve (an irregular,~o:^|^^) drawn with a light solid line represents the 
area of the &e^ph wji^e the density of public schools is the highest--, 
over 100 schools per s.quare unit of achievement (i.e., one correct v 
question on the r^adi^g test); ' .The second solid closed curve represents 
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°the- area o{ next. highest density, with over 50 schools per square unit. 

t * ' 

SiK hundred seventy-five schools lie in these two areas. Finally the third 
• ^ and largest ifregu-Iar oval encloses, most of the remaining data points for 
public schools. In this area, .158 schools are shown by dots. Outside of 
this third irregular oval are the 19 outliers. Superimposed on this are 
data points for 82 Catholic schools, shown as solid circles, surrounded ty 
a heayy curve 'which enclosed all of them. Finally, the 28 non-Catholic \ 
scKools are shown .by +'s and enclosed , by a double line. The straight line 
in the' figure is the llrife- generated by the' regression equation. 

• j'' ; ____•„ ' • 

^gure 1 A)Out Here 

— — ~~ ~ ~~ , 

' t 

^ ^ The plot shows few irregularities. Some of the lowest scores are 

contributed by non-Catholic private schools. These may be schools ^ 
catering to students ^with certain types of learning or behavior prob-lems . 
Parochial schools lie slightly above the regres^sion line, but sino^theif 
mean is only .34 units above the "expected, the difference is not easily 
.discerned ia the plot. There .is a- slight curvilinearity appearing ia' the 
: plot—it appears that the best fitting line would be concave upward (i.e., 

^ a positive derivative throughout). A- regression of tie predictor polynomial 

plus its. square does fits the 'data slightly* better , but has no important 
^ Imbact on the. results.. (In the quadratic equations, one of the private 

^ " • ^. ' ' ' ^ 

' school coefficients becomes significantly negative, and the parochial ^ 

^ I 

school coefficiehts drop slightly.) The plot does illustrate-^visually 



the major problem with the attempt to measure the impact of private 

* * *» * . 

schools: the higher social status of the student bodies. Of the 82 

Catholic schools, 36 have a predi'Cted achievement level or 8 units o^^^ 
higher.' Seventeen of the 2A private schaols have exfjected achievement. 
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levels this high, while only 14% of all public schools f.ll in this ra^ge 

I . « ■ ■ 

_ Conclusions 

. Aggregation of data to the school or classfoojn leyeJ^ ,^hould clearly ' 
Be limited to situations where the independent variable interest, is 
itse^lf a school or'classroom characteristic^. But this is often the case 
in educational researcj^; and wheo. it is, d.ata aggregation is useful to ' 
reduce wha^ig'ht otherwise , be an unmanageable mass of data. Because It 
does not solve the problem of self-selection, and only reduces rather 
than eliminates problems of attenuation through measurement. error , we' do 
not *ant to exagerate its value. 
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Footnotes 



1. Tffiere is a good deal written on this: The clearest expositions are 
in numerous papers by Donald Campbell. for example, Campbell 
and Erlebacker (1970). For a discussion of attenuation, see Guilford 
(1954). ' * * 

2. There has been much Crit'icism,. most of it as yet unpublished. Society 
is preparing a symposium, on the report, and a paper in that volume ' 
(forthcoming) reviews many of the critiques. See also Educational 
Research Service, 1981. ' , ^ f 

3. This conclusion is Mrawn by both Richard Murnane and David .Krathwohl 
in .their critiques appearing in Educational Researc^h Service, 1981. 

4. The data and* codebooks are available from the National Center for 
Education Statistics. ' * 

5. 'students in these schools were given very small weights, so that when 

a weighted analysis i$ made, the elite schools are essentially dis- 
carded. ^ . . 

6. For a lengthy analysis done almo^st entirely with schooj-level equations 
and equations' which Jhiix school^^d individual-level data, see Grain, 
Mah^-rd, and Narot (1^82). <J ' ^ 
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Table X: Individual* and Aggregate Level Analysis with* 
' on^ Control Variable 



Dependent Variable 



Vocabulary 



ratio, 



bontrolled b 
uncontrolled b 



Reading 



ratio 



Math 



^ controlled b' 
'uncontrolled b 



ratio 



controlled b 
'uncontrolled b 



Level 




Regression Coefficients 




' Catholic ^^'^^'^ 

Private 


Father's 
Education 


Constant 


Multiple 
R 


Individual 
Aggregate 


• 2.722 

^ • 0 j4 


2.981 
3.136 


t 


o, U14 
8.015 • 


.156 
.327 


Individual 
Aggregate 


1.943 
.994 


1.675 
.299 


' .633 / 
1.435 


5.375 
2.080 


.359 
.735 


Individual 
Aggregate 


.72 
.37 


■' .56 
-. .10 






• 


Individual 
Aggregate 


1.940 


1.882 
•1*981 




0 . DO J 

6.565, 


.118 
.296 


Individual 
Aggregate 


' 1.336 
.720 


\ .868 
-.017 


.492 
1.011 


4. 512 
2.385 


.330 
.683 " 


Individual ' 
Aggregate 


.38 


. 46 

0 




• 


• 


Individual 
Aggregate 


> 

3.455 
3.367 


3.883 
.r^091 • 




9.630 ■ 
9.625 


.136 
.298 


' Individual 
Aggregate 


2.389 
1.128 


2.096 
.265 


.867 • 
1.935 


6.017 
1.6^ 


330 • 
.696 


Individuifr 

Aggregate 

> 


.69 
.34 ■ 


\ .54 - 
. .06 








* 


t 
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Table 2^ 



Individual and Aggregate Equations, with Full Set of Control Variables 



Independent Variables , 
Catholic • 
Non-Catjiolic Private 
j89 . Slack 
90 . Hispaijiic 
39.' Father's Education 
Mother's Education 
Family Income „^ 
Rooms in Home 
Number of Siblings 
Own: Calculator ^ ' 
50+ Books 
^ J Encyclopedia 
Typewriter- 
Father 

50B. for Child: Mother 
Mother Before e.s. 
worked: During e.'^. 
Father In Home^ 
Mother in Home 
T^lk with Parents 
Constant 
Multiple r 
n, unweighed 
n, weighted 
Ratio,. 

Controlled b . 



42. 
101. 
103. 

41. 
4G. 
4C. 
4D, 

50A. Aspirations 
for Chi 

37C. 
37B. 
38B. 
36D. 
47G. 



Unstan^ardized Regression Coefficients 



'vocabulary 



reading 



X 

a 



inc^iv 

8.276 
5.232 



1.14 



.62 
. 3.34* 
. • -2.03* 
.20* 
.26* 
.14 
.09 
-.21* 
.88* 
.90* 
.11 
.17 
1.03* 
'.99 
-.26 , 
-.05 
-.12^ 
.42 
.20 
11.77 

.550 
30,263 
3,422,479 



agg 

8.254 
2.540 



Uncontrolled b . 
Ratios from pps, 
jlable 6.2.1 , 



Catholic 42% 

Non-Catholic 21% 

Catholic . 44% 

Non-Catholic 20% 



indiv 

6.742 
4.73 



.50*1 
-.14 
r3.15* 
-1.89* 
. .25*1 
.47*1 
.24*1 
.18*'| 
-.17*^ 
1.14* 
1.77* 
.36 
i 1.54* 

.25 
i 1.97*i 
; '-.90 
-.49 
-.63^ 
1.-89* 
i -.13^ 
1 -2.15 
' .868 
1 973 



]9% 



agg 

6.712 
1.909 



.70 
.08 
■2.31* 

•;l,.88*| 

.15* , 
.20* ; 
.04 ■ 
.08 
/.14* ' 
.74* ' 
.62 
.24 ! 
-.02^ 
1.03* I 
1.04* ' 
•-.27 
.01^ 
.03 
.46 I 
.23 ! 
7.63 I 
.478 1 
30,263 I 
3,421,054i 

( 

1 

36% 1 
4% { 
40% 
33% 



math 



indiv 



.965 
.686 



.34 
-T% 
-2.05* 
-1.98* 
.18* 
.34* 
-.02a 
.Z5* 
-.14* 
.88* 
1.03* 
.41 
' .51 
.20 
1.81^^ 
-.72 
-.01 
.49 
1.70 
.33" 
-1.82 
.677 
973 



18% 
0% 



1.09 
.68 
-4.47* 
-3.18* 
.2i3* 
.27* 
.18 
.23 
-.18* 
1.27* 
.78 ■ 
, .16 
.37 
1.96* 
2.06* 
-.52 • 
.02^ 
.24 
.96 
.17 
10.52' 
.529 
30,263 
3,412,433 

32% 
18% 
38% 
28% 



agg 

9.919 
3.585 



.19 
V.36 
-3.59* 
s-2.64* 
.23* 
.63* ^ 
. .26 
.56* 
-.15 
2.17* 
.86 
..15' 
1.93* 
-.18a' 

4.15^ 
-.81 
-1.23 
1.35 
3.25* 
-.26^ 
-7,51 
.847 
973 



5% 
0'% 



p < .05 (see text) 



^sign in unexpected direction 
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